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Summary 

The  Visual  Threat  Recognition  and  Avoidance  Trainer  (VTRAT)  system  addresses  the  need  for  realistic  training  that 
enables  forces  to  respond  quickly  and  operate  effectively  during  threat  engagements.  This  initiative  combines  state-of- 
the-art  visual,  interactive  simulations  with  intelligent  tutoring  methodologies  to  train  the  visual  scanners.  VTRAT  is  an 
automated  virtual  intelligent  instructional  training  aid,  designed  to  introduce  or  refresh  scanners  on  visual  recognition 
of  threats  and  on  their  duties  during  a  threat  engagement.  This  paper  describes  two  studies  that  employ  a  backward 
transfer  procedure  for  validating  the  training  effectiveness  of  VTRAT.  In  experiment  1,  34  Air  Force  aircrew 
personnel  were  tested  at  Air  Force  Special  Operations  Command  on  their  identification  of  surface-to-air  missiles 
(SAMs).  Several  types  of  SAMs  were  simulated  and  presented  to  participants.  Participants  identified  the  model  of 
SAM,  and  the  accuracy  of  trajectory  (whether  or  not  the  SAM  would  hit  their  simulated  aircraft).  Participants  were 
assigned  to  either  a  novice  group  (no  combat  experience)  or  expert  group  (combat  experience  in  either  the  Kosovo  or 
Iraqi  air  campaigns).  The  experts  outperformed  the  novices  on  all  performance  measures.  In  experiment  2,  39  aircraft 
scanners  with  the  19th  Special  Operations  Squadron  at  Hurlbert  Field,  FL  completed  VTRAT  training  upon  return 
from  deployment  in  Afghanistan.  There  was  a  significant  positive  correlation  between  number  of  antiaircraft  artillery 
(AAA)  threats  observed  in  theater  and  AAA  threat  recognition  in  VTRAT.  Together  these  two  studies  provide 
evidence  supporting  the  validity  of  VTRAT  training  for  recognizing  and  avoiding  visual  threats.  Additionally,  survey 
results  indicated  that  scanners  rated  the  VTRAT  visuals  to  be  highly  realistic  and  VTRAT  training  to  be  valuable 
compared  to  previous  training  methods. 

Introduction 

Air  Force  Special  Operations  Command  (AFSOC)  aircraft  scanners  (crewmembers)  have  the  primary  duty  to 
identify  antiaircraft  threats,  direct  the  pilot  and  crew  in  performance  of  evasive  maneuvers,  and  deploy 
countermeasures  during  an  antiaircraft  threat  engagement.  Survival  of  slow-moving  aircraft  during  enemy 
engagements  with  non-radar  antiaircraft  weapons  hinges  on  the  rapid  detection  and  responses  of  visual  scanners.  Once 
enemies  engage  using  optically  guided  antiaircraft  artillery  or  infrared  surface-to-air  missiles  (IR-SAM),  scanners 
must  detect  the  threat  and  respond  with  appropriate  verbal  calls  dispensing  countermeasures  within  a  matter  of 
seconds. 

Traditionally  aircrew  members  have  received  little  or  no  training  in  detecting  threats.  Operating  procedures  are 
rarely  practiced.  It  is  impractical  to  train  detection  skills  by  firing  live  rounds  or  actual  SAMs  at  aircraft  during 
training  flights.  Moreover,  while  procedures  for  handling  threat  engagements  are  written  down,  they  are  not  practiced 
often  enough  to  become  routine.  In  fact,  many  crewmembers  may  not  be  completely  familiar  with  the  specified 
procedures.  Consequently,  when  facing  threats  in  actual  operations,  crewmembers  may  not  have  the  ability  to  respond 
in  an  accurate  and  timely  manner. 


Paper  presented  at  the  RTO  HFM  Symposium  on  “ Advanced  Technologies  for  Military  Training  ”, 
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In  1998  the  Warfighter  Training  Research  Division  of  the  Air  Force  Research  Laboratory  (AFRL/HEAI),  began 
development  of  a  synthetic  trainer  for  the  19th  Special  Operations  Squadron  (SOS).  This  training  system,  called  the 
Visual  Threat  Recognition  and  Avoidance  Trainer  (VTRAT),  was  designed  to  train  loadmasters  in  AC-130  Gunships 
to  rapidly  detect  and  execute  countermeasures  for  various  simulated  threats  (Figure  1).  Basic  system  functionality, 
dictated  by  the  prescribed  training  methodology,  includes:  (a)  a  view  of  the  threat  environment  from  the  perspective  of 
the  scanner  in  the  airframe,  (b)  accurate  visualization  of  threat  characteristics,  (c)  a  high-fidelity  interface  for  scanner 
communications  and  countermeasures,  and  d)  dynamic  feedback  indicating  timeliness,  accuracy,  and  effectiveness  of 
response. 


Figure  1.  Visual  Threat  Recognition  and  Avoidance  Trainer  (VTRAT). 

The  system  combines  state-of-the-art  visual,  interactive  simulations  with  intelligent  tutoring  methodologies  to 
provide  an  adaptive  and  efficient  solution  for  training  visual  scanners.  VTRAT  decomposes  the  skill  of  threat  detection 
and  avoidance  into  subskills  and  focuses  on  each  subskill.  First  the  visual  characteristics  of  various  threats  are 
displayed  and  discussed.  Then  learners  are  trained  to  recognize  and  distinguish  enemy  launches  that  pose  an  actual 
threat  to  the  aircraft  from  those  that  do  not.  After  learners  achieve  a  level  of  proficiency  on  recognizing  threats,  the 
avoidance  procedures  are  discussed  and  illustrated.  Following  this  discussion,  learners  have  an  opportunity  to  practice 
these  avoidance  procedures.  In  this  way  the  skill  is  built  up  from  declarative  knowledge  into  automated  behavior. 

Due  to  the  success  of  the  initial  project,  the  VTRAT  system  was  expanded  to  include  all  visual  scanners  on  the 
AC-130  (Gunship),  MC-130/U  (Talon  I),  MC130/H  (Talon  II),  MC130/P  (Shadow),  and  the  MH-53  (Pavelow).  This 
expanded  project  required  coding  for  each  additional  crew  position  viewpoint.  In  addition,  each  airframe  was 
programmed  to  fly  slightly  different  profiles  and  each  was  programmed  for  slightly  different  avoidance  tactics. 
Moreover,  the  threat  systems  being  modeled  were  certified  on  visual  characteristics  of  antiaircraft  weaponry,  including 
missile  fly-out,  AAA  rate-of-fire,  trajectory,  speed,  etc.,  by  the  Air  Force  Information  Warfare  Center  as  valid  for 
training  purposes. 

The  goal  of  this  research  project  is  to  validate  the  training  effectiveness  of  the  VTRAT  system  for  training  visual 
scanners  to  recognize  threats  and  execute  avoidance  procedures.  The  most  optimal  way  to  test  the  validity  is  to 
measure  the  transfer  from  the  training  system  to  actual  operations  or  actual  exercise  with  live  fire.  Cost,  risk,  and 
logistics  render  this  method  of  validation  infeasible. 

An  alternative  approach  to  validation  is  to  measure  the  task  transfer  from  experts  to  the  training  system. 
Knowledge  and  skill  transfer  is  not  unidirectional.  Expertise  gained  during  training  transfers  to  the  relevant  task,  but 
expertise  gained  on  the  relevant  task  will  also  transfer  back  to  training.  The  critical  assumption  is  that  the  fidelity  of 
the  training  to  the  task  is  sufficient  to  promote  transfer.  Thus  the  representation  of  the  necessary  task  knowledge  must 
be  sufficiently  analogous  to  the  task  to  promote  transfer  (e.g.,  see  Singley  &  Anderson,  1989;  Gick  &  Holyoak,  1980, 
1983;  Holyoak,  1984). 
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Measuring  the  extent  to  which  skills  and  expertise  acquired  in  field  operations  transfer  back  to  training  systems 
provides  a  training  validation  metric.  This  procedure  is  called  backward  transfer.  The  amount  of  backward  transfer 
should  reflect  the  fidelity  of  the  training  system  (Wightman  &  Lintem,  1985).  Goettl  and  Shute  (1996)  demonstrated 
how  backward  transfer  could  be  used  to  improve  the  efficiency  of  part-task  training.  The  backward  transfer  procedure 
is  depicted  in  Figure  2.  In  their  study  Goettl  and  Shute  trained  one  group  using  whol-task  training  and  another  group 
using  a  series  of  part  tasks.  Following  training  both  groups  performed  the  whole  task  and  then  completed  the  part 
tasks.  We  compared  the  performance  of  the  whole-task  group  on  the  part  tasks  to  the  initial  performance  of  the  part- 
task  group  on  the  part  tasks.  Any  part  tasks  that  the  whole-task  group  performed  better  (i.e.,  showed  backward 
transfer)  were  identified  as  critical  tasks  and  were  included  in  a  modified  part-task  condition.  The  modified  part-task 
condition  resulted  in  overall  better  training  than  whole -task  training. 


Experts 


ice^^ 


Novices 


Figure  2.  Backward  Transfer  approach  to  training  validation. 

In  the  present  study  backward  transfer  will  be  used  to  measure  the  training  validity  of  the  VTRAT  system.  The 
approach  was  similar  to  that  employed  by  Goettl,  Ashworth,  and  Chaiken  (in  press).  Modular  Control  Equipment 
(MCE)  operators  conduct  the  same  functions  as  Airborne  Warning  and  Control  System  (AWACS)  weapons  directors 
except  from  ground-based  stations.  These  duties  and  responsibilities  include:  control  friendly  fighter  aircraft,  monitor 
friendly  fuel  consumption,  refuel  friendly  assets,  monitor  hostile  aircraft,  protect  ground  assets,  deploy  fighters  against 
hostiles  when  necessary,  collaborate  with  other  operators,  pass  control  of  aircraft  back  and  forth  between  sectors  of  the 
airspace.  Goettl  et  al.  had  Air  Force  Modular  Control  Equipment  (MCE)  operators  perform  a  synthetic  team  task 
configured  to  simulate  the  duties  and  responsibilities  of  a  team  of  AWACS  weapons  directors.  Operators  performed 
the  task  under  two  different  scenarios  and  with  two  different  interfaces.  Goettl  et  al.  compared  the  operators’ 
performance  during  acquisition  and  transfer  phases  with  the  performance  of  novices.  They  found  that  operators 
performed  better  during  acquisition  and  transfer  phases  of  the  task.  While  novices  showed  dramatic  performance 
decrements  when  the  interface  was  changed,  experienced  operators  showed  no  decrement  at  all.  This  clearly  indicates 
that  the  operators  had  a  richer  knowledge  of  the  AWACS  task  and  were  able  to  transfer  that  knowledge  to  the  training 
task  regardless  of  changes  in  the  scenario  and  interface. 

The  present  study  presents  two  backward  transfer  experiments  to  validate  the  VTRAT  system  for  training 
effectiveness.  In  Experiment  1  we  compare  the  VTRAT  performance  of  self-identified  threat  experts  against  the 
performance  of  threat  novices.  The  prediction  is  that  experts  will  perform  better  on  VTRAT.  In  Experiment  2  the 
performance  on  VTRAT  of  aircrews  will  be  correlated  with  the  amount  of  threat  experience  during  recent  operations 
including  Kosovo  and  Afghanistan.  A  positive  correlation  reflects  positive  transfer  and  will  provide  evidence  of 
training  efficacy.  Aircraft  scanners  from  the  19th  SOS  at  Flurlbert  Field,  FL.  completed  VTRAT  training  upon  return 
from  deployment  in  Afghanistan.  Correlations  were  calculated  between  experience  with  real-world  threat  engagements 
and  VTRAT  performance.  In  addition,  we  administered  surveys  to  assess  judgments  of  simulation  realism  and  training 
effectiveness. 
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Experiment  1 


ORGANIZATION 


Method 

Subjects.  Thirty-four  Air  Force  Aircrew  personnel  were  tested  at  AFSOC  at  Flurlbert  Field,  FF.  Participants  were 
assigned  to  either  a  Novice  (n  =  25)  or  Expert  (n  =  9)  group  based  on  their  real-world  military  combat  experience.  The 
Novice  group  were  Special  Operations  trainees,  none  of  whom  had  combat  experience.  The  Expert  group  had  C-130 
experience  during  either  Operation  Allied  Force  (Kosovo)  or  Operation  Desert  Storm  (Iraq),  and  had  engaged  actual 
enemy  air  defenses. 

Apparatus  and  Materials.  The  VTRAT  is  a  state-of-the-art  training  system  that  incorporates  high-fidelity  graphics 
and  voice  recognition  technologies  (Figure  1).  Five  software  components  collaborate  to  provide  VTRAT  functionality: 
an  Image  Generator  (IG),  an  Interaction  Server  (IS),  a  Threat  System  (TS),  an  Aircraft  Host  (AH),  and  a  Situated 
Adaptive  beaming  Technology  (SAFT)  component.  These  components  were  implemented  using  commercial-off-the- 
shelf  (COTS)  and  government-off-the-shelf  (GOTS)  development  tools  on  COTS  hardware. 

The  SAFT  component  allows  both  the  automated  instruction  and  an  instructor  to  control  the  simulated 
environment.  The  instructor  uses  SAFT’s  graphical  user  interface  (GUI)  to  control  the  simulated  threat  environment 
and  monitor  student  performance.  The  GUI  allows  the  instructor  to  modify  numerous  simulation  dynamics  such  as 
scanner  viewpoint,  aircraft  altitude,  airspeed.  Currently,  VTRAT  contains  courses  to  train  threat  recognition  and 
avoidance  for  all  duty  positions  requiring  scanner  training  in  the  Gunship,  Talon-I,  Talon-II,  Shadow,  and  MH-53J 
weapon  systems. 

The  IG,  TS,  and  AH  form  the  heart  of  the  3-D  visual  simulation.  These  components  run  on  an  SGI  Onyx2-IR2  and 
are  certified  High  Fevel  Architecture  (HFA)  compliant.  The  aircraft  host  models  a  C-130  or  MH-53J,  on  a  mission 
flight  path,  dynamically  changing  altitude,  airspeed,  and  flight  path  as  requested  through  scanner  communications,  the 
instructor’s  GUI,  or  automated  instruction.  The  TS  models  a  variety  of  AAA  and  SAMs.  It  dynamically  generates 
threats  based  on  instructional  objectives  that  dictate  threat  placement  and  targeting  parameters. 

The  student  views  the  appropriate  crew  position  perspective  of  the  simulated  threat  environment  on  a  high- 
resolution  67”  display  system.  The  student  wears  a  headset,  with  a  microphone  attached  for  purposes  of  voice 
recognition,  and  holds  a  communication  control  and  a  flares  countermeasures  control.  Both  controls  are  from  the 
actual  aircraft,  modified  to  communicate  with  VTRAT  through  a  PC  game  port.  The  IS  (running  Windows  2000  on  a 
500  MHz  PC)  delivers  instructional  text  as  synthesized  speech  output,  and  monitors  scanner  communications.  The 
speech  recognition  software  provides  speaker-independent,  continuous  speech  recognition,  with  no  training 
requirements. 

Procedure.  In  the  first  phase  trainees  were  required  to  discriminate  between  the  three  SAMs  until  they  reached  an 
accuracy  criterion.  They  also  had  to  state  whether  or  not  the  SAM  was  accurate  or  inaccurate  (would  or  would  not  hit 
the  C-130).  There  was  no  time  pressure.  Both  the  identity  and  accuracy  of  the  SAMs  were  evenly  distributed  across 
trials.  To  successfully  complete  this  phase  trainees  had  to  complete  at  least  20  trials,  and  9  of  the  last  10  trials  had  to 
be  correct  for  both  identity  and  accuracy  of  SAM.  Trainee’s  verbal  responses  were  processed  by  the  speech 
recognition  system,  which  then  supplied  accuracy  feedback.  The  second  phase  was  identical  to  the  first  except  that 
it  was  conducted  under  time  pressure.  The  trainees  had  to  meet  criterion  first  within  an  8-second  time 
window,  then  a  6-second  window,  and  finally  within  a  4-second  window.  Thus  not  responding  within  the 
time  window  became  an  additional  type  of  error. 

Results 

Reported  here  are  the  data  from  the  final  phase  of  training.  Independent  samples  t-tests  revealed  four  main  effects 
(Table  1).  Due  to  unequal  group  size,  homogeneity  of  variance  was  not  assumed.  There  was  a  significant  difference  in 
Proportion  Correct,  t  (12.58)  =  2.49,  p  <  .05,  indicating  that  Experts  (M  =  .78)  had  a  higher  proportion  correct  than 
Novices  (M  =  .68).  There  was  a  significant  difference  in  total  number  of  SAMs  shown,  t  (26.69)  =  -2.22,  p  <  .05, 
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indicating  that  Novices  (M  =  32.28)  required  more  trials  to  reach  criterion  than  Experts  (M  =  24.33).  There  was  a 
significant  difference  in  the  number  of  Time  Outs,  t  (17.02)  =  -2.05,  p  =  .05,  indicating  that  Novices  (M  =  5.44)  were 
more  likely  to  miss  the  time  window  for  responses  than  were  Experts  (M  =  2.89).  Finally,  there  was  a  significant 
difference  in  the  overall  minutes  to  finish  training,  t  (31.09)  =  -2.00,  p  =  .05,  indicating  that  Novices  took  longer  (M  = 
16.56)  than  Experts  (M  =  12.59). 


Dependent 

Variable 

Novice 

Expert 

Significance 

Proportion 

Correct 

.68 

.78 

<.05 

Total  Number 
Shown 

32.28 

24.33 

<.05 

Time  Outs 

5.44 

2.89 

<.05 

Total  Minutes 
to  Criterion 

16.56 

12.59 

<.05 

Table  1.  Significant  VTRAT  outcome  measures  with  means. 


Discussion 

The  superior  performance  of  the  Expert  group  suggests  that  experts  are  in  fact  transferring  their  extant  knowledge 
from  their  real-world  SAM  experience  to  the  VTRAT  training  task.  It  can  then  be  inferred  that  there  is  sufficient 
fidelity  between  VTRAT  and  its  real-world  counterpart  to  make  VTRAT  a  viable  training  device. 

It  is  often  the  case  that  training  systems  cannot  be  validated  through  their  real-world  counterparts.  Elowever,  this 
does  not  preclude  other  methods  of  validation.  Measuring  the  transfer  from  experts’  extant  knowledge  back  to  the 
training  system  is  an  affordable  and  theoretically  appropriate  method  of  validation  in  such  circumstances. 

Considering  the  high  costs  in  human  life,  materiel,  and  mission  effectiveness  associated  with  ineffective  training 
systems,  there  is  ample  reason  why  Air  Force  training  systems  such  as  VTRAT  should  always  be  validated. 

Experiment  2 


Method 

Participants.  Participants  were  39  Aircrew  members  from  the  19th  and  20th  Air  Force  SOSs.  Participants  ranged  in 
age  from  18  to  30  and  had  a  high  school  diploma  or  Graduate  Equivalency  Degree  (GED).  Participants  included  pilots, 
co-pilots,  flight  engineers,  loadmasters,  scanners,  and  gunners  from  five  different  airframes:  gunship,  Talon  I,  Talon  II, 
shadow,  and  MEI-53.  All  participants  were  recruited  by  AFSOC  personnel  and  participated  voluntarily. 

Equipment.  The  equipment  used  in  Experiment  2  is  the  same  as  was  used  in  Experiment  1. 

VTRAT  Task.  In  this  Experiment  participants  completed  two  drill  lessons  of  VTRAT.  In  one  drill  participants  had 
to  detect  and  initiate  avoidance  tactics  for  different  IR  SAM  threats.  The  threats  modeled  in  the  system  were  three 
MANPAD  systems  including  the  SA-7,  SA-14,  and  SA-16.  Participants  had  to  make  the  appropriate  call  and  deploy 
countermeasures  within  a  specified  length  of  time.  The  drill  consisted  of  20  trials.  In  the  second  drill,  participants  had 
to  detect  and  initiate  avoidance  tactics  for  AAA  threats.  Participants  were  graded  on  each  component  of  the  avoidance 
call  and  whether  they  completed  the  call  within  a  specified  window.  As  part  of  the  detection,  participants  had  to 
determine  whether  the  AAA  threats  were  firing  at  the  aircraft  or  not,  and  whether  the  maneuver  they  called  was 
appropriate.  Participants  completed  a  minimum  of  20  trials  and  had  to  get  9  of  the  last  1 0  to  complete  the  drill.  Five 
different  AAA  threats  were  modeled:  14.5  mm,  23-2mm,  23-4mm,  37mm,  and  57mm. 

Questionnaire.  The  questionnaire  consisted  of  three  parts.  The  first  part  of  the  survey  assessed  general  mission 
experience  as  well  as  combat  experience.  Participants  indicated  number  of  flights  and  total  flight  hours  in  theatre.  In 
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addition  they  were  asked  to  indicate  how  many  times  they  had  observed  SAM  and  AAA  threats  firing  at  them  or  other 
aircraft  in  theater. 


The  second  part  of  the  survey  dealt  with  specific  threats  modeled  in  VTRAT.  Participants  were  asked  to  indicate 
how  often  they  had  observed  each  of  the  different  threats  modeled  in  VTRAT  and  to  rate  the  VTRAT  simulations  on 
different  dimensions.  Rating  scales  consisted  of  a  6-point  Likert-type  scale  with  anchors  representing  very  inaccurate 
(1)  and  very  accurate  (6).  Only  those  having  seen  the  specific  threats  in  theater  were  required  to  rate  the  simulations. 

In  the  final  section  of  the  survey  participants  were  asked  to  rate  the  quality  of  VTRAT  training  compared  to 
previous  training  methods.  Participants  rated  the  quality  of  the  discussion  of  the  visuals,  discussion  of  the  tactics,  and 
realism  of  the  threats  and  scenarios.  Ratings  were  made  using  a  6-point  Likert-type  scale  with  poor  (1)  and  excellent 
(6)  as  the  anchors.  The  aspects  of  VTRAT  that  were  rated  included  (a)  discussion  of  visual  features  of  AAA  threats, 
(b)  discussion  of  visual  features  of  IR-SAM  threats,  (c)  discussion  of  avoidance  tactics  for  AAA  threats,  (d)  discussion 
of  avoidance  tactics  for  IR-SAM  threats,  (e)  realism  of  AAA  threats,  and  (f)  realism  of  IR-SAM  threats. 

Procedure.  The  data  reported  here  were  collected  at  the  19th  SOS  in  Hurlbert  Field,  FL.  Participants  signed  in  and 
were  given  a  subject-matter  expert  (SME)  number  to  maintain  confidentiality.  They  were  briefed  on  the  goals  and 
purposes  of  the  study.  Following  these  instructions,  participants  completed  the  IR-SAM  recognition  and  avoidance 
drill.  Next,  participants  completed  the  AAA  threat  avoidance  drill.  Finally,  participants  were  asked  to  complete  the 
questionnaire,  were  debriefed,  and  released. 

Results 

Of  the  39  participants  only  33  completed  the  drills.  Due  to  computer  errors,  data  from  3  participants  were  lost.  Of 
the  remaining  30  participants  with  performance  data,  there  were  16  gunship  crewmembers,  3  Talon  crewmembers,  6 
Shadow  crewmembers,  and  5  MH-53  crewmembers. 

Performance  and  Experience.  The  primary  goal  of  this  study  was  to  validate  the  accuracy  of  the  simulation.  The 
approach  was  to  correlate  accuracy  of  threat  calls  with  amount  of  experience.  If  the  visual  characteristics  of  the  threat 
simulations  and  flight  simulation  are  accurate,  there  should  be  a  significant  positive  correlation  between  performance 
on  the  threat  calls  and  amount  of  experience.  In  the  present  study,  experience  was  measured  multiple  ways:  number  of 
missions  in  theater,  number  of  hours  in  theater,  and  number  of  times  AAA  or  SAMs  were  seen  fired  upon  self  or 
others.  On  average,  participants  viewed  AAA  shot  at  themselves  3.67  time  (SD  =  6.12),  AAA  fired  at  others  3.07 
times  (SD  =  3.859),  SAMs  shot  at  themselves  0.33  times  (SD  =  0.84),  and  SAMs  fired  at  others  0.23  times  (SD  = 
0.73). 

Each  of  these  measures  was  correlated  with  performance  on  threat.  Table  2  shows  the  correlations  between 
experience  measures  and  accuracy  of  AAA  and  SAM  threat  calls  and  locations. 


Experience 

AAA 

Threat 

AAA 

Location 

SAM 

Threat 

SAM 

Location 

Total 

Missions 

.155 

.036 

-.128 

-.132 

Total  Hours 

.057 

-.276 

.133 

-.053 

Threat  at  own 
ship 

.116 

.227 

.109 

-.042 

Threat  at 
other  ships 

.447* 

.301 

.184 

-.102 
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Table  2.  Correlations  between  VTRAT  performance  and  threat  experience:  number  of  missions,  number  of  mission 
hours,  number  of  times  viewed  threat  fired  at  own  aircraft,  and  number  of  times  viewed  AAA  fired  at  other  aircraft  (*p 
<  .05). 


As  indicated  in  Table  2,  in  general,  performance  on  AAA  threat  calls  in  VTRAT  are  positively  correlated  with 
experience.  Moreover,  correlations  tend  to  be  stronger  for  the  experience  metrics  based  on  frequency  of  viewing  actual 
threats  in  theater.  In  spite  of  this  positive  evidence,  only  the  correlation  between  threat  call  and  number  of  times 
viewed  AAA  fired  at  others  reached  statistical  significance. 

Correlations  involving  SAM  performance  and  experience,  as  indicated  in  Table  2  were  not  as  strong  as 
correlations  for  AAA.  This  is  likely  due  to  a  restriction  of  range  problem.  That  is,  very  few  participants  had  viewed 
actual  SAM  threats  in  theater.  Additionally,  performance  on  SAM  threat  calls  were  highly  accurate  ranging  from  95% 
correct  for  shadow  crews  to  98  %  correct  for  gunship  crews.  Nevertheless  the  correlations  between  threat  call  accuracy 
and  the  number  of  times  SAMs  had  been  viewed  are  positive. 

Subjective  Ratings  of  Training  Value.  An  important  metric  in  training  is  the  perceived  value  of  training. 
Participants  were  asked  to  compare  the  VTRAT  training  to  previous  training  approaches  on  several  dimensions.  These 
dimensions  included  discussion  of  visuals  and  tactics  for  AAA  and  SAM  threats,  and  realism  of  AAA  and  SAM 
threats.  Table  3  shows  the  average  ratings  for  several  VTRAT  dimensions. 


VTRAT  Dimension 

Average  Rating 

Discussion  of  AAA  Visuals 

5.48 

Discussion  of  SAM  Visuals 

5.46 

Discussion  of  AAA  Tactics 

5.04 

Discussion  of  SAM  Tactics 

5.35 

Realism  of  AAA  Threats 

5.27 

Realism  of  SAM  Threats 

5.24 

Table  3.  Average  ratings  of  VTRAT  quality  metrics 

As  indicated  in  Table  3,  overall  the  ratings  of  VTRAT  were  relatively  high.  On  a  6-point  scale  where  6  is  excellent,  no 
rated  aspect  of  VTRAT  was  rated  below  a  5.00.  Compared  to  other  training  approaches  that  participants  experienced, 
VTRAT  was  rated  nearly  excellent  on  all  dimensions.  The  discussion  of  visuals  showed  the  highest  ratings  followed 
by  the  discussion  of  SAM  tactics,  then  realism  of  the  threats.  The  aspect  of  VTRAT  rated  the  lowest  was  the 
discussion  of  AAA  tactics.  This  is  not  entirely  surprising  given  that  avoidance  of  AAA  threats  is  more  complex  and 
based  on  perceptual  judgments  that  are  difficult  to  model  on  a  2-dimensional  display. 

Discussion 

The  data  presented  here  support  the  conclusion  that  VTRAT  provides  valuable  training  experience  to  visual 
scanners.  This  evidence  comes  not  only  from  the  subjective  ratings  of  visual  scanners  but  from  backward  transfer 
analysis. 

Perhaps  the  most  important  finding  in  this  study  is  the  strong  positive  correlation  between  experience  and  VTRAT 
performance.  Accuracy  of  AAA  threat  calls  was  positively  and  significantly  correlated  with  the  number  of  times 
scanners  had  viewed  AAA  threats  firing  at  other  aircraft.  This  is  the  kind  of  backward  transfer  evidence  that  supports 
the  validity  of  training.  Those  with  more  experience  and  familiarity  with  actual  threats  performed  better  on  recognition 
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and  avoidance  drills,  W41e  in  the  current  data,  the  strongest  effects  were  shown  for  AAA  performance,  correlations 
between  SAM  threat  c  and  experience  were  positive  for  all  experience  measures  except  total  missions. 


The  difference  in  backward  transfer  between  AAA  experience  and  SAM  experience  may  reflect  the  relatively  little 
experience  participants  had  with  SAM  threats.  Moreover,  the  experience  with  SAM  engagements  was  not  correlated  to 
total  number  of  missions  (.116  and  -.050  for  SAMs  at  you  and  at  others,  respectively).  For  AAA  experience  the 
number  of  total  missions  was  strongly  correlated  with  number  AAA  threats  viewed  (.626  and  .517  for  AAA  fired  at 
you  and  at  others,  respectively).  Again  the  weak  correlations  likely  reflect  restriction  of  range  problems.  Very  few 
participants  had  seen  SAMs  at  the  time  of  testing  (M  =  .33). 

Finally,  we  obtained  strong  positive  results  from  the  subjective  ratings  of  the  training  value.  All  rated  aspects  of 
VTRAT  training  received  an  average  rating  of  5  or  more  on  a  scale  where  6  represented  excellent  value.  These  data 
reflect  the  general  response  that  VTRAT  developers  have  received  from  AFSOC  personnel.  In  fact,  the  system  was  so 
highly  valued  that  AFSOC  personnel  created  compact  disks  of  VTRAT  simulations  and  passed  them  around  to 
aircrews  during  the  Afghanistan  operations.  In  summary,  the  data  presented  demonstrate  and  confirm  the  validity  of 
VTRAT  simulations  and  drills  for  training  the  recognition  and  avoidance  of  visual  threats. 

General  Discussion 

These  two  experiments  provide  converging  evidence  that  the  VTRAT  system  is  effective  for  training  the 
recognition  and  avoidance  of  threats.  Both  experiments  used  a  backward  transfer  approach  to  training  validation.  This 
technique  measures  the  amount  of  transfer  from  the  real-world  task  to  the  training  system.  This  approach  assumes  that 
if  the  training  system  captures  important  task  dimensions,  then  experts  will  perform  better  on  the  training  system  than 
will  novices.  In  Experiment  1  backward  transfer  was  measured  by  comparing  VTRAT  performance  of  operators  with 
combat  experience  to  performance  of  operators  without  combat  experience.  In  that  experiment  experts  performed 
better  on  all  performance  metrics  of  IR-SAM  detection  in  the  VTRAT.  This  finding  supports  the  conclusion  that 
experts  were  able  to  transfer  combat  experience  to  the  training  environment. 

In  Experiment  2,  experience  was  measured  more  directly  through  a  survey  measuring  specific  experience 
perceiving  AAA  and  SAM  threats.  In  this  experiment,  the  extent  of  real-world  experience  viewing  AAA  was 
positively  correlated  with  VTRAT  performance  in  detecting  AAA  threats.  The  finding  that  operators  with  greater 
experience  viewing  actual  AAA  threats  performed  better  than  those  with  less  experience  supports  the  conclusion  that 
VTRAT  captures  important  dimensions  of  the  threat  detection  task. 

It  is  difficult  to  validate  the  effectiveness  of  simulators  designed  for  combat  training.  The  most  logical  approach 
for  measuring  transfer  from  the  training  to  the  operational  task  is  not  practical.  Measuring  transfer  from  the  training 
system  to  a  very  high-fidelity  simulation  can  be  expensive  and  is  limited  to  the  validity  of  the  simulation.  The 
backward  transfer  approach  provides  a  practical  inexpensive  way  of  validating  the  effectiveness  of  military  training 
systems.  The  present  paper  demonstrates  two  approaches  to  backward  transfer  to  the  validation  of  a  training  system 
and  can  serve  as  a  model  for  future  validation  efforts. 
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